Information retrieval utilizing semantic representation of text by identifying hypernyms and indexing multiple tokenized semantic structures to a same passage of text

ABSTRACT

The present invention is directed to performing information retrieval utilizing semantic representation of text. In a preferred embodiment, a tokenizer generates from an input string information retrieval tokens that characterize the semantic relationship expressed in the input string. The tokenizer first creates from the input string a primary logical form characterizing a semantic relationship between selected words in the input string. The tokenizer then identifies hypernyms that each have an &#34;is a&#34; relationship with one of the selected words in the input string. The tokenizer then constructs from the primary logical form one or more alternative logical forms. The tokenizer constructs each alternative logical form by, for each of one or more of the selected words in the input string, replacing the selected word in the primary logical form with an identified hypernym of the selected word. Finally, the tokenizer generates tokens representing both the primary logical form and the alternative logical forms. The tokenizer is preferably used to generate tokens for both constructing an index representing target documents and processing a query against that index.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of U.S. patent application No.08/886,814, filed Mar. 7, 1997, now U.S. Pat. No. 6,076,051.

TECHNICAL FIELD

The present invention relates to the field of information retrieval,and, more specifically, to the field of information retrievaltokenization.

BACKGROUND OF THE INVENTION

Information retrieval refers to the process of identifying occurrencesin a target document of words in a query or query document. Informationretrieval can be gainfully applied in several situations, includingprocessing explicit user search queries, identifying documents relatingto a particular document, judging the similarities of two documents,extracting the features of a document and summarizing a document.

Information retrieval typically involves a two-stage process: (1) In anindexing stage, a document is initially indexed by (a) converting eachword in the document into a series of characters intelligible to anddifferentiable by an information retrieval engine, called a "token"(known as "tokenizing" the document) and (b) creating an index mappingfrom each token to the location in the document where the token occurs.(2) In a query phase, a query (or query document) is similarly tokenizedand compared to the index to identify locations in the document at whichtokens in the tokenized query occur.

FIG. 1 is an overview data flow diagram depicting the informationretrieval process. In the indexing stage, a target document 111 issubmitted to a tokenizer 112. The target document is comprised of anumber of strings, such as sentences, each occurring at a particularlocation in the target document. The strings in the target document andtheir word locations are passed to a tokenizer 120, which converts thewords in each string into a series of tokens that are intelligible toand distinguishable by an information retrieval engine 130. An indexconstruction portion 131 of the information retrieval engine 130 addsthe tokens and their locations to an index 140. The index maps eachunique token to the locations at which it occurs in the target document.This process may be repeated to add a number of different targetdocuments to the index, if desired. If the index 140 thus represents thetext in a number of target documents, the location informationpreferably includes an indication of, for each location, the document towhich the location corresponds.

In the query phase, a textual query 112 is submitted to the tokenizer120. The query may be a single string, or sentence, or may be an entiredocument comprised of a number of strings. The tokenizer 120 convertsthe words in the text of the query 112 into tokens in the same mannerthat it converted the words in the target document into tokens. Thetokenizer 120 passes these tokens to an index retrieval portion 132 ofthe information retrieval engine 130. The index retrieval portion of theinformation retrieval engine searches the index 140 for occurrences ofthe tokens in the target document. For each of the tokens, the indexretrieval portion of the information retrieval engine identifies thelocations at which the token occurs in the target document. This list oflocations is returned as the query result 113.

Conventional tokenizers typically involve superficial transformations ofthe input text, such as changing each upper-case character tolower-case, identifying the individual words in the input text, andremoving suffixes from the words. For example, a conventional tokenizermight convert the input text string

The father is holding the baby.

into the following tokens:

the

father

is

hold

the

baby

This approach to tokenization tends to make searches based on itoverinclusive of occurrences in which senses of words are different thanthe intended sense in the query text. For example, the sample input textstring uses the verb "hold" in the sense that means "to support orgrasp." However, the token "hold" could match uses of the word "hold"that mean "the cargo area of a ship." This approach to tokenization alsotends to be overinclusive of occurrences in which the words relate toeach other differently than the words in the query text. For example,the sample input text string above, in which "father" is the subject ofthe word "held" and "baby" is the object, might match the sentence "Thefather and the baby held the toy," in which "baby" is a subject, not anobject. This approach is further underinclusive of occurrences that usea different, but semantically related word in place of a word of thequery text. For example, the input text string above would not match thetext string "The parent is holding the baby." Given these disadvantagesof conventional tokenization, a tokenizer that enacts semanticrelationships implicit in the tokenized text would have significantutility.

SUMMARY OF THE INVENTION

The invention is directed to performing information retrieval using animproved tokenizer that parses input text to identify logical forms,then expands the logical forms using hypernyms. The invention, when usedin conjunction with conventional information retrieval indexconstruction and querying, reduces the number of identified occurrencesfor which different senses were intended and in which words beardifferent relationships to each other, and increases the number ofidentified occurrences in which different but semantically related termsare used.

The invention overcomes the problems associated with conventionaltokenization by parsing both indexed and query text to perform lexical,syntactic, and semantic analysis of this input text. This parsingprocess produces one or more logical forms, which identify words thatperform primary roles in the query text and their intended senses, andthat further identify the relationship between those words. The parserpreferably produces logical forms that relate the deep subject, verb,and deep object of the input text. For example, for the input text "Thefather is holding the baby," the parser might produce the followinglogical form:

    ______________________________________                                        deep subject    verb   deep object                                            ______________________________________                                        father          hold   baby                                                   ______________________________________                                    

The parser further ascribes to these words the particular senses inwhich they are used in the input text.

Using a digital dictionary or thesaurus (also known as a "linguisticknowledge base") that identifies, for a particular sense of a word,senses of other words that are generic terms for the sense of the word("hypernyms"), the invention changes the words within the logical formsproduced by the parser to their hypernyms to create additional logicalforms having an overall meaning that is hypernymous to the meaning ofthese original logical forms. For example, based on indications from thedictionary that a sense of "parent" is a hypernym of the ascribed senseof "father," a sense of "touch" is a hypernym of the ascribed sense of"hold," and a sense of "child" and sense of "person" are hypernyms ofthe ascribed sense of "baby," the invention might create additionallogical forms as follows:

    ______________________________________                                        deep subject    verb   deep object                                            ______________________________________                                        parent          hold   baby                                                   father          touch  baby                                                   parent          touch  baby                                                   father          hold   child                                                  parent          hold   child                                                  father          touch  child                                                  parent          touch  child                                                  father          hold   person                                                 parent          hold   person                                                 father          touch  person                                                 parent          touch  person                                                 ______________________________________                                    

The invention then transforms all of the generated logical forms intotokens intelligible by the information retrieval system that comparesthe tokenized query to the index, and submits them to the informationretrieval system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overview data flow diagram depicting the informationretrieval process.

FIG. 2 is a high-level block diagram of the general-purpose computersystem upon which the facility preferably operates.

FIG. 3 is an overview flow diagram showing the steps preferablyperformed by the facility in order to construct and access an indexsemantically representing the target documents.

FIG. 4 is a flow diagram showing the tokenize routine used by thefacility to generate tokens for an input sentence.

FIG. 5 is a logical form diagram showing a sample logical form.

FIG. 6 is an input text diagram showing an input text fragment for whichthe facility would construct the logical form shown in FIG. 5.

FIG. 7A is a linguistic knowledge base diagram showing sample hypernymrelationships identified by a linguistic knowledge base.

FIG. 7B is a linguistic knowledge base diagram showing the selection ofhypernyms of the deep subject of the primary logical form, man (sense2).

FIG. 8 is a linguistic knowledge base diagram showing the selection ofhypernyms of the verb of the primary logical form, kiss (sense 1).

FIGS. 9 and 10 are linguistic knowledge base diagrams showing theselection of hypernyms of the deep object of the primary logical form,pig (sense 2).

FIG. 11 is a logical form diagram showing the expanded logical form.

FIG. 12 is a chart diagram showing the derivative logical forms createdby permuting the expanded primary logical form.

FIG. 13 is an index diagram showing sample contents of the index.

FIG. 14 is a logical form diagram showing the logical form preferablyconstructed by the facility for the query "man kissing horse."

FIG. 15 shows the expansion of the primary logical form using hypernyms.

FIG. 16 is a linguistic knowledge base diagram showing the selection ofhypernyms of the deep object of the query logical form, horse (sense 1).

FIG. 17 is a partial logical form diagram showing a partial logical formcorresponding to a partial query containing only a deep subject and averb.

FIG. 18 is a partial logical form diagram showing a partial logical formcorresponding to a partial query containing only a verb and a deepobject.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is directed to performing information retrievalutilizing semantic representation of text. When used in conjunction withconventional information retrieval index construction and querying, theinvention reduces the number of identified occurrences for whichdifferent senses were intended and in which words bear differentrelationships to each other, and increases the number of identifiedoccurrences in which different but semantically related terms are used.

In a preferred embodiment, the conventional tokenizer shown in FIG. 1 isreplaced with an improved information retrieval tokenization facility("the facility") that parses input text to identify logical forms, thenexpands the logical forms using hypernyms. The invention overcomes theproblems associated with conventional tokenization by parsing bothindexed and query text to perform lexical, syntactic, and semanticanalysis of this input text. This parsing process produces one or morelogical forms, which identify words that perform primary roles in thequery text and their intended senses, and that further identify therelationship between those words. The parser preferably produces logicalforms that relate the deep subject, verb, and deep object of the inputtext. For example, for the input text "The father is holding the baby,"the parser might produce logical form indicating the deep subject is"father," the verb is "hold," and the deep object is "baby." Becausetransforming input text into a logical form distills the input text toits fundamental meaning by eliminating modifiers and ignoringdifferences in tense and voice, transforming input text segments intothe logical forms tends to unify the many different ways that may beused in a natural language to express the same idea. The parser furtheridentifies the particular senses of these words in which they are usedin the input text.

Using a digital dictionary or thesaurus (also known as a "linguisticknowledge base") that identifies, for a particular sense of a word,senses of other words that are generic terms for the sense of the word("hypernyms"), the invention changes the words within the logical formsproduced by the parser to their hypernyms to create additional logicalforms having an overall meaning that is hypernymous to the meaning ofthese original logical forms. The invention then transforms all of thegenerated logical forms into tokens intelligible by the informationretrieval system that compares the tokenized query to the index, andsubmits them to the information retrieval system.

FIG. 2 is a high-level block diagram of the general-purpose computersystem upon which the facility preferably operates. The computer system200 contains a central processing unit (CPU) 210, input/output devices220, and a computer memory (memory) 230. Among the input/output devicesis a storage device 221, such as a hard disk drive. The input/outputdevices also include a computer-readable media drive 222, which can beused to install software products, including the facility which areprovided on a computer-readable medium, such as a CD-ROM. Theinput/output devices further include an Internet connection 223 enablingthe computer system 200 to communicate with other computer systems viathe Internet. The computer programs that preferably comprise thefacility 240 reside in the memory 230 and execute on the CPU 210. Thefacility 240 includes a rule-based parser 241 for parsing input textsegments to be tokenized in order to produce logical forms. The facility240 further includes a linguistic knowledge base 242 used by the parserto ascribe sense numbers to words in the logical form. The facilityfurther uses the linguistic knowledge base to identify hypernyms of thewords in the generated logical forms. The memory 230 preferably alsocontains an index 250 for mapping from tokens generated from the targetdocuments to locations in the target documents. The memory 230 alsocontains an information retrieval engine ("IR engine") 260 for storingtokens generated from the target documents in the index 250, and foridentifying in the index tokens that match tokens generated fromqueries. While the facility is preferably implemented on a computersystem configured as described above, those skilled in the art willrecognize that it may also be implemented on computer systems havingdifferent configurations.

FIG. 3 is an overview flow diagram showing the steps preferablyperformed by the facility in order to construct and access an indexsemantically representing the target documents. Briefly, the facilityfirst semantically indexes the target documents by converting eachsentence or sentence fragment of the target document into a number oftokens representing an expanded logical form portraying the relationshipbetween the important words in the sentence, including hypernyms havingsimilar meanings. The facility stores these "semantic tokens" in theindex, along with the location in the target documents where thesentence occurs. After all of the target documents have been indexed,the facility is able to process information retrieval queries againstthe index. For each such query received, the facility tokenizes the textof the query in the same way it tokenized sentences from the targetdocuments--by converting the sentence into semantic tokens togetherrepresenting an expanded logical form for the query text. The facilitythen compares these semantic tokens to the semantic tokens stored in theindex to identify locations in the target documents for which thesesemantic tokens have been stored, and ranks the target documentscontaining these semantic tokens in the order of their relevance to thequery. The facility may preferably update the index to include semantictokens for new target documents at any time.

Referring to FIG. 3, in steps 301-304, the facility loops through eachsentence in the target documents. In step 302, the facility invokes aroutine to tokenize the sentence as shown in FIG. 4.

FIG. 4 is a flow diagram showing the tokenize routine used by thefacility to generate tokens for an input sentence or other input textsegment. In step 401, the facility constructs a primary logical formfrom the input text segment. As discussed above, a logical formrepresents the fundamental meaning of a sentence or sentence fragment.The logical forms are produced by applying the parser 241 (FIG. 2) tosubject the input text segment to a syntactic and semantic parsingprocess. For a detailed discussion of the construction of logical formsrepresenting an input text string, refer to U.S. patent application No.08/674,610, which is hereby incorporated by reference.

The logical form used by the facility preferably isolates the principalverb of the sentence, the noun that is the real subject of the verb("deep subject") and the noun that is the real object of the verb ("deepobject"). FIG. 5 is a logical form diagram showing a sample primarylogical form. The logical form has three elements: a deep subjectelement 510, a verb element 520, and a deep object element 530. It canbe seen that the deep subject of the logical form is sense 2 of the word"man." The sense number indicates, for words having more than one sense,the particular sense ascribed to the word by the parser as defined bythe linguistic knowledge base used by the parser. For example, the word"man" could have a first sense meaning to supply with people and asecond sense meaning adult male person. The verb of the logical form isa first sense of the word "kiss." Finally, the deep object is a secondsense of the word "pig." An abbreviated version of this logical form isan ordered triple 550 having as its first element the deep subject, asits second element the verb, and as its third element the deep object:

(man, kiss, pig)

The logical form shown in FIG. 5 characterizes a number of differentsentences and sentence fragments. For example, FIG. 6 is an input textdiagram showing an input text segment for which the facility wouldconstruct the logical form shown in FIG. 5. FIG. 6 shows the input textsentence fragment "man kissing a pig." It can be seen that this phraseoccurs at word number 150 of document 5, occupying word positions 150,151, 152, and 153. When the facility is tokenizing this input textfragment, it generates the logical form shown in FIG. 5. The facilitywould also generate the logical form shown in FIG. 5 for the followinginput text segments:

The pig was kissed by an unusual man.

The man will kiss the largest pig.

Many pigs have been kissed by that man.

As discussed above, because transforming input text into a logical formdistills the input text to its fundamental meaning by eliminatingmodifiers and ignoring differences in tense and voice, transforminginput text segments into the logical forms tends to unify the manydifferent ways that may be used in a natural language to express thesame idea.

Returning to FIG. 4, after the facility has constructed the primarylogical form from the input text, such as the logical form shown in FIG.5, the facility continues in step 402 to expand this primary logicalform using hypernyms. After step 402, the tokenized routine returns.

As mentioned above, a hypernym is a genus term that has an "is a"relationship with a particular word. For instance, the word "vehicle" isa hypernym of the word "automobile." The facility preferably uses alinguistic knowledge base to identify hypernyms of the words in theprimary logical form. Such a linguistic knowledge base typicallycontains semantic links identifying hypernyms of a word.

FIG. 7A is a linguistic knowledge base diagram showing sample hypernymrelationships identified by a linguistic knowledge base. It should benoted that FIG. 7A, like the linguistic knowledge base diagrams thatfollow, has been simplified to facilitate this discussion, and omitsinformation commonly found in linguistic knowledge bases that is notdirectly relevant to the present discussion. Each ascending arrow inFIG. 7A connects a word to its hypernym. For example, there is an arrowconnecting the word man (sense 2) 711 to the word person (sense 1) 714,indicating that person (sense 1) is a hypernym of man (sense 2).Conversely, man (sense 2) is said to be a "hyponym" of person (sense 1).

In identifying hypernyms with which to expand the primary logical form,the facility selects one or more hypernyms for each word of the primarylogical form based upon the "coherency" of the hypernyms' hyponyms. Byselecting hypernyms in this manner, the facility generalizes the meaningof the logical form beyond the meaning of the input text segment, but bya controlled amount. For a particular word of a primary logical form,the facility first selects the immediate hypernym of the word of theprimary logical form. For example, with reference to FIG. 7A, startingwith man (sense 2) 711 which occurs in the primary logical form, thefacility selects its hypernym, person (sense 1) 714. The facility nextbases its determination of whether to also select the hypernym of person(sense 1) 714, animal (sense 3) 715, on whether person (sense 1) 714 hasa coherent hyponym set with respect to the starting word man (sense 2)711. Person (sense 1) 714 has a coherent hyponym set with respect to man(sense 2) 711 if a large number of hyponyms of all senses of the wordperson other than the starting word (sense 2) 711 bear at least athreshold level of similarity to the starting word man (sense 2) 711.

In order to determine the level of similarity between the hyponyms ofthe different senses of the hypernym, the facility preferably consultsthe linguistic knowledge base to obtain similarity weights indicatingthe degree of similarity between these word sentences. FIG. 7B is alinguistic knowledge base diagram showing similarity weights between man(sense 2) and other hyponyms of person (sense 1) and person (sense 5).The diagram shows that the similarity weight between man (sense 2) andwoman (sense 1) is "0.0075"; between man (sense 2) and child (sense 1)is "0.0029"; between man (sense 2) and villain (sense 1) is "0.0003";and between man (sense 2) and lead (sense 7) is "0.0002". Thesesimilarity weights are preferably calculated by the linguistic knowledgebase based on a network of semantic relations maintained by thelinguistic knowledge base between the word sense pairs. For a detaileddiscussion of calculating similarity weights between word sense pairsusing a linguistic knowledge base, refer to U.S. patent application No.08/904,223, entitled "DETERMINING SIMILARITY BETWEEN WORDS," which ishereby incorporated by reference.

In order to determine whether the set of hyponyms is coherent based onthese similarity weights, the facility determines whether a thresholdnumber of the similarity weights exceed a threshold similarity weight.While the preferred threshold percentage is 90%, the thresholdpercentage may preferably be adjusted in order to optimize theperformance of the facility. The similarity weight threshold may also beconfigured to optimize the performance of the facility. The thresholdsimilarity weight is preferably coordinated with the overalldistribution of similarity weights provided by the linguistic knowledgebase. Here, the use of a threshold of "0.0015" is shown. The facilitytherefore determines whether at least 90% of the similarity weightsbetween the starting word and the other hyponyms of all of the senses ofthe hypernym are at or above the "0.0015" threshold similarity weight.It can be seen from FIG. 7B that this condition is not satisfied by thehyponyms of person with respect to man (sense 1): while the similarityweights between man (sense 1) and woman (sense 1) and between man(sense 1) and child (sense 1) are greater than "0.0015", the similarityweights between man (sense 1) and villain (sense 1) and between man(sense 1) and lead (sense 7) are less than "0.0015". The facilitytherefore does not select the further hypernym animal (sense 3) 715, orany hypernyms of animal (sense 3). As a result, only the hypernym person(sense 1) 714 is selected to expand the primary logical form.

To expand a primary logical form, the facility also selects hypernyms ofthe verb and deep object of the primary logical form. FIG. 8 is alinguistic knowledge base diagram showing the selection of hypernyms ofthe verb of the primary logical form, kiss (sense 1). It can be seenfrom the diagram that touch (sense 2) is the hypernym of kiss (sense 1).The diagram also shows the similarity weights between kiss (sense 1) andthe other hyponyms of all of the senses of touch. The facility firstselects the immediate hypernym of the verb of the primary logical formkiss (sense 1), touch (sense 2). To determine whether to select thehypernym of touch (sense 2), interact (sense 9), the facility determineshow many similarity weights between kiss (sense 1) and the otherhyponyms of all of the senses of touch are at least as large as thethreshold similarity weight. Because only two of these four similarityweights are at least as large as the "0.0015" threshold similarityweight, the facility does not select the hypernym of touch (sense 2),interact (sense 9).

FIGS. 9 and 10 are linguistic knowledge base diagrams showing theselection of hypernyms of the deep object of the primary logical formand pig (sense 2). It can be seen from FIG. 9 that the facility selectsthe hypernym swine (sense 1) of pig (sense 2) to expand the primarylogical form, as well as the hypernym animal (sense 3) of swine (sense1), as more than 90% (in fact, 100%) of the hypernyms of the only senseof swine have similarly weights at or about the "0.0015" thresholdsimilarity weight. It can be seen from FIG. 10 that the facility doesnot continue to select the hypernym organism (sense 1) of animal (sense3), as fewer than 90% (actually 25%) of the hyponyms of senses of animalhave similarity weights at or about the "0.0015" threshold similarityweight.

FIG. 11 is a logical form diagram showing the expanded logical form. Itcan be seen from FIG. 11 that the deep subject element 1110 of theexpanded logical form contains the hypernym person (sense 1) 1112 inaddition to the word man (sense 2) 1111. It can be seen that the verbelement 1120 contains the hypernym touch (sense 2) 1122 as well as theword kiss (sense 1) 1121. Further, it can be seen that the deep objectelement 1130 of the expanded logical form contains the hypernyms swine(sense 1) and animal (sense 3) 1132 in addition to the word pig (sense2) 1131.

By permuting, in each element of the expanded logical form, thehypernyms with the original words, the facility can create a reasonablylarge number of derivative logical forms that are reasonably close inmeaning to the primary logical form. FIG. 12 is a chart diagram showingthe derivative logical forms created by permuting the expanded primarylogical form. It can be seen from FIG. 12 that this permutation createseleven derivative logical forms that each characterize the meaning ofthe input text in a reasonably accurate way. For example, the derivativelogical form

(person, touch, pig)

shown in FIG. 12 is very close in meaning to the sentence fragment

man kissing a pig

The expanded logical form shown in FIG. 11 represents the primarylogical form plus these eleven derivative logical forms, which areexpressed more compactly as expanded logical form 1200:

((man OR person), (kiss OR touch), (pig OR swine OR animal))

The facility generates logical tokens from this expanded logical form ina manner that allows them to be processed by a conventional informationretrieval engine. First, the facility appends a reserved character toeach word in the expanded logical form that identifies whether the wordoccurred in the input text segment as a deep subject, verb, or deepobject. This ensures that, when the word "man" occurs in the expandedlogical form for a query input text segment as a deep subject, it willnot match the word "man" stored in the index as part of an expandedlogical form in which it was the verb. A sample mapping of reservedcharacters to logical form elements is as follows:

    ______________________________________                                        logical form element                                                                         identifying character                                          ______________________________________                                        deep subject   .sub.--                                                        verb                                                                          deep object    #                                                              ______________________________________                                    

Using this sample mapping of reserved characters, tokens generated forthe logical form "(man, kiss, pig)" would include "man₋₋ ", "kiss ", and"pig#".

Indices generated by conventional information retrieval engines commonlymap each token to the particular locations in the target documents atwhich the token occurs. Conventional information retrieval engines may,for example, represent such target document locations using a documentnumber, identifying the target document containing the occurrence of thetoken, and a word number, identifying the position of the occurrence ofthe token in that target document. Such target document locations allowa conventional information retrieval engine to identify words that occurtogether in a target document in response to a query using a "PHRASE"operator, which requires the words that it joins to be adjacent in thetarget document. For example, the query "red PHRASE bicycle" would matchoccurrences of "red" at document 5, word 611 and "bicycle" at document5, word 612, but would not match occurrences of "red" at document 7,word 762 and "bicycle" at document 7, word 202. Storing target documentlocations in an index further allows conventional information retrievalengines to identify, in response to a query, the points at which queriedtokens occur in the target documents.

For expanded logical forms from a target document input text segment,the facility preferably similarly assigns artificial target documentlocations to each token, even though the tokens of the expanded logicalform do not actually occur in the target document at these locations.Assigning these target document locations both (A) enables conventionalsearch engines to identify combinations of semantic tokens correspondingto a single primary or derivative logical form using the PHRASEoperator, and (B) enables the facility to relate the assigned locationsto the actual location of the input text fragment in the targetdocument. The facility therefore assigns locations to semantic tokens asfollows:

    ______________________________________                                        logical form element                                                                           location                                                     ______________________________________                                        deep subject     (location of 1st word of input                                                text segment)                                                verb             (location of 1st word of input                                                text segment) + 1                                            deep object      (location of 1st word of input                                                text segment) + 2                                            ______________________________________                                    

The facility therefore would assign target document locations as followsfor the tokens of the expanded logical form for "(man, kiss, pig)",derived from a sentence beginning at document 5, word 150: "man₋₋ " and"person₋₋ "--document 5, word 150; "kiss " and "touch "--document 5,word 151; and "pig#", "swine#", and "animal#"--document 5, word 152.

Returning to FIG. 3, in step 303, the facility stores the tokens createdby the tokenize routine in the index with locations at which they occur.FIG. 13 is an index diagram showing sample contents of the index. Theindex maps from each token to the identity of the document and locationin the document at which they occur. It should be noted that, while theindex is shown as a table to more clearly show the mappings in theindex, the index is actually preferably stored in one of a number ofother forms that support more efficient location of a token in theindex, such as in tree form. Further, the contents of the index arepreferably compressed to minimize the size of the index, using suchtechniques as prefix compression.

It can be seen that, in accordance with step 303, the facility hasstored mappings in the index 1300 for each of the words in the expandedlogical form. Mappings have been stored in the index from deep subjectwords "man" and "person" to the target document location at documentnumber 5, word number 150. Word number 150 is the word position at whichthe input text segment shown in FIG. 6 begins. It can be seen that thefacility has appended the reserved character "₋₋ " to the tokenscorresponding to the deep subject words. By appending this reservedcharacter, the facility is able to retrieve, when later searching theindex, instances of these words that occur as the deep subject of alogical form without retrieving occurrence of these words that occur asverbs or deep objects of a logical form. Similarly, the index containstokens for verb words "kiss" and "touch." The entries for these verbwords map them to the target document location at document number 5,word number 151, one word after the target document location of the deepsubject words. It can further be seen that the reserved character " "has been appended to the tokens for these verb words so that thisoccurrence of these words does not later appear to be an occurrence as adeep subject or deep object element. Likewise, the index contains tokensfor the deep object words "animal," "pig," and "swine," mapping them tothe target document location at document number 5, word number 152, twowords past the target document location at which the phrase begins. Thereserved character "#" is appended to the tokens for the deep objectwords to identify them as deep objects in the index. With the index inthe condition shown, the input text fragment shown in FIG. 6 can befound by searching the index for any of the derivative primary logicalforms shown in FIG. 12.

In a preferred embodiment in which the facility stores both a mapping ofthe words literally occurring in the target documents to their actuallocations of the target documents and the semantic representation of thetarget documents in the same index, the word number values for eachsemantic token of the semantic representation is preferably incrementedby a constant larger than the number of words in any document todistinguish semantic tokens of the semantic representation from literaltokens when accessed in the index. To simplify FIG. 13, the addition ofthis constant is not shown.

In the example, the facility adds a token for each of the words in theexpanded logical form to the index to form the semantic representationof the target documents. In one preferred embodiment, however, thefacility limits the set of expanded logical form tokens that it adds tothe index to those logical form tokens that are likely to be effectiveat distinguishing between documents among the target documents. To solimit the set of expanded logical form tokens added to the index, thefacility preferably determines the Inverse Document Frequency of eachtoken, whose formula is shown by equation (1) below. In this embodiment,the facility adds to the index only tokens whose Inverse DocumentFrequency exceeds a minimum threshold.

Returning to FIG. 3, after storing the tokens in the index before thecurrent sentence in the target document, in step 304, the facility loopsback to step 301 to process the next sentence in the target documents.When all of the sentences of the target document have been processed,the facility continues at step 305. In step 305, the facility receivesthe text of a query. In steps 306-308, the facility processes thereceived query. In step 306, the facility invokes the tokenized routineto tokenize the query text. FIG. 14 is a logical form diagram showingthe logical form preferably constructed by the facility for the query"man kissing horse" in accordance with step 401 (FIG. 4). It can be seenfrom the logical form diagram that the deep subject is man (sense 2),the verb is kiss (sense 1), and the deep object is horse (sense 1). Thisprimary logical form is more succinctly represented as

(man, kiss, horse)

in primary logical form 1450.

FIG. 15 shows the expansion of the primary logical form using hypernymsin accordance with steps 402 (FIG. 4). It can be seen from FIG. 15 that,like the sample input text from the target document, the deep subjectman (sense 2) has been expanded with the hypernym person (sense 1), andthe verb kiss (sense 1) has been expanded with the hypernym touch (sense2). Further, it can be seen that the deep object horse (sense 1) hasbeen expanded with hypernym animal (sense 3).

FIG. 16 is a linguistic knowledge base diagram showing the selection ofhypernyms of the deep object of the query logical form, horse (sense 1).It can be seen from FIG. 16 that the facility does not select thehypernym organism (sense 1) of animal (sense 3), since fewer than 90% ofthe hyponyms of animal (sense 3) have similarity weights at or above the"0.0015" threshold similarity weight. The facility therefore uses onlythe hypernym animal (sense 3) to expand the logical form.

Returning to FIG. 3, in step 307, the facility uses the expanded logicalform 1550 (FIG. 15) constructed using hypernyms of the word senses inthe primary logical form to retrieve from the index locations in thetarget documents at which matching tokens occur. The facility preferablydoes so by issuing the following query against the index:

(man₋₋ OR person₋₋ PHRASE (kiss OR touch ) PHRASE (horse™ OR animal#)

The PHRASE operator matches occurrences of the operand following it at aword position 1 greater than the operand preceding it. Therefore, thequery matches where the deep subject man₋₋ or person₋₋ precedes the verbkiss or touch , which precedes the deep object horse# or animal#. It canbe seen from the index in FIG. 13 that this query is satisfied atdocument number 5, word number 150.

If this query was not satisfied in the index, the facility wouldcontinue to submit the query in two different partial queries. The firstpartial form contains only the deep subject and the verb, and not theobject:

(man₋₋ OR person₋₋) PHRASE (kiss OR touch )

FIG. 17 is a partial logical form diagram showing the partial logicalform corresponding to this first query. The second partial form of thequery contains the verb and deep object, but not the deep subject:

(kiss OR touch ) PHRASE (horse# OR animal#)

FIG. 18 is a partial logical form diagram showing the partial logicalform corresponding to this second partial query. These partial querieswould match logical forms in the index having a different deep subjector deep object, and would match partial logical forms not having a deepsubject or deep object. These partial queries take into considerationdifferences between the query input text segment and target documentinput text segments including pronoun usage and implied deep subjectsand deep objects.

Returning to FIG. 3, after identifying matches of tokens in the index,the facility continues in step 308 to rank the target documents in whichmatches of particular combinations of matching tokens, corresponding toa primary or derivative logical form, occur in the order of theirrelevance to the query. In various embodiments of the invention, thefacility employs one or more of a number of well-known approaches toranking documents by relevancy, which include Jaccard weighting andbinary term independence weighting. The facility preferably uses acombination of inverse document frequency and term frequency waiting torank the matching target documents.

The inverse document frequency weight characterizes a tokencombination's ability to distinguish between documents, giving greaterweight to a token combination appearing in fewer of the targetdocuments. For example, for a group of target documents directed to thesubject of photography, the logical form

(photographer, frame, subject)

could occur in each document of the group, and thus would not be a verygood basis for distinguishing between documents. Because the abovelogical form occurs in every target document, it has a relatively smallinverse document frequency. The formula for inverse document frequencyof a token combination is as follows: ##EQU1##

The term frequency weight of a token combination in a document measuresthe extent to which the document is dedicated to the token combination,and assumes that a document in which a particular query token occurs alarge number of times is more relevant than a document in which thequery token occurs fewer times. The formula for the term frequencyweight of a token combination in document is as follows:

    Term Frequency (token combination, document)=number of times token combination occurs in document                            (2)

The facility uses a score for each matching document to rank thedocuments. The facility first calculates a score for each matching tokencombination in each document, using the following formula:

    Score (token combination, document)=Inverse Document Frequency (token combination)×Term Frequency (token combination, document) (3)

The facility then calculates the score for each matching document bychoosing the highest score for any matching token combination in eachmatching document in accordance with the following formula: ##EQU2##Once the facility has calculated a score for each document, the facilitymay augment these scores to reflect terms of the query other than thosedirected to semantic matching. After augmenting the score for eachdocument, if necessary, the facility calculates a normalized score foreach document by taking the size of the document into account as shownin the following formula: ##EQU3## The Size (document) term may be anyreasonable measure of the size of a document--for example, the number ofcharacters, words, or sentences or sentence fragments in the document.The document score may alternatively be normalized using a number ofother normalization techniques, including cosine measure normalization,sum of term weights normalization, and maximum term weightsnormalization.

After calculating normalized scores for each matching document, thefacility ranks the matching documents in order of the normalized scoresfor the documents. A user may preferably select one of the matchingdocuments from the ranked list to obtain a location of the matchingtokens in that document, or to display the matching portion of thatdocument.

Returning to FIG. 3, after ranking the matching target documents in step308, the facility preferably continues at step 305 to receive the textof the next query against the index.

The above discusses ranking by relevancy the documents containingmatching tokens. Additional preferred embodiments of the inventionsimilarly rank by relevancy document groups and document sections,respectively, that contain matches. For target documents that areorganized into document groups each containing one or more documents,the facility preferably ranks the document groups in which matches occurby relevancy in order to identify the most relevant document groups forfurther querying. Further, the facility is preferably configurable todivide each target document into sections and rank the relevancy ofdocument sections in which matches occur. These document sections may beidentified contiguously within a target document either by selecting acertain number of bytes, words, or sentences, or by using structural,formatting, or linguistic cues occurring in the target document. Thefacility may also preferably identify non-contiguous document sectionsdealing with particular themes.

While the present invention has been shown and described with referenceto preferred embodiments, it will be understood by those skilled in theart that various changes or modifications in form and detail may be madewithout departing from the scope of the invention. For example, thetokenizer may be straightforwardly adapted to produce and store in theindex tokens each corresponding to a complete logical form constructioninstead of tokens each corresponding to one word of a logical formconstruction. Also, various well-known techniques may be applied toincorporate other types of searching in a query having a semanticmatching component. Further, a query may contain a number of semanticmatching components. In addition, semantic relationships identifiedbetween words other than hypernyms may be used to expand the primarylogical form. The facility may also use precompiled lists ofsubstitutable words for each word in a primary logical form to expandthe primary logical form, rather than generating lists of hypernyms froma lexical knowledge base at runtime as described above. Further, foradditional matching precision, the tokenizer may encode in the token fora word the sense number identified for the word. In this case, the testfor coherency of the hyponym set is reduced from testing similarity withall senses of the selected hypernym. In the example, only the hyponymsof sense 1 of the word person need to bear a threshold level ofsimilarity with the starting sense of the word man (sense 2). Becausethe possible matching terms in the index are less ambiguous we canconstrain the set of terms which might produce false hits. For thisreason it is only necessary to test for the senses which have a hypernymrelation to the word in the logical form.

What is claimed is:
 1. A method in a computer system for generatinginformation retrieval tokens from an input string, the method comprisingthe steps of:creating from the input string a primary logical formcharacterizing a semantic relationship between selected words in theinput string; identifying hypernyms of the selected words in the inputstring, at least one hypernym being identified from a group of hypernymsassociated with a selected word wherein at least one other hypernym inthe group is not identified; constructing from the primary logical formone or more alternative logical forms, each alternative logical formbeing constructed by, for each of one or more of the selected words inthe input string, replacing the selected word in the primary logicalform with an identified hypernym of the selected word; and generatingtokens representing both the primary logical form and the alternativelogical forms, the generated tokens being distinguishable by aninformation retrieval engine.
 2. The method of claim 1 wherein theconstructing step includes the step of parsing the input string todiscern its syntactic and semantic structure.
 3. The method of claim 2wherein the identifying step includes the steps of:for each selectedword of the input text:retrieving from a linguistic knowledge base oneor more hypernyms of the selected word each having a similarity valuecharacterizing the similarity in meaning of the hypernym to the selectedword; and identifying any of the hypernyms whose similarity valueexceeds a preestablished threshold.
 4. The method of claim 1, furthercomprising the steps of:before the constructing step, selecting theinput string from a search query; and submitting the generated tokens toa query engine for comparison to a representation of one or more targetdocuments.
 5. The method of claim 1, further comprising the stepsof:before the constructing step, selecting the input string from a bodyof text to be indexed; and submitting the generated tokens to anindexing subsystem for storage in an index representing the body oftext.
 6. The method of claim 5, further including the step ofdetermining an inverse document frequency of each of the words occurringin the alternative logical forms, and wherein the submitting step omitsto submit to the indexing subsystem tokens representing alternativelogical forms containing words whose inverse document frequency issmaller than a preestablished minimum inverse document frequency.
 7. Themethod of claim 5, further including the steps of:after the submittingstep, determining an inverse document frequency of each of the wordsoccurring in the alternative logical forms; and removing from the indextokens representing alternative logical forms containing words whoseinverse document frequency is smaller than a preestablished minimuminverse document frequency.
 8. The method of claim 1 wherein theidentifying step identifies hypernyms of the selected words that havecoherent hyponym sets with respect to the selected words.
 9. Acomputer-readable medium whose contents cause a computer system togenerate information retrieval tokens from an input string by performingthe steps of:creating from the input string a primary logical formcharacterizing a semantic relationship between selected words in theinput string; identifying hypernyms of the selected words in the inputstring, at least one hypernym being identified from a group of hypernymsassociated with a selected word wherein at least one other hypernym inthe group is not identified; constructing from the primary logical formone or more alternative logical forms, each alternative logical formbeing constructed by, for each of one or more of the selected words inthe input string, replacing the selected word in the primary logicalform with an identified hypernym of the selected word; and generatingtokens representing both the primary logical form and the alternativelogical forms, the generated tokens being distinguishable by aninformation retrieval engine.
 10. The computer-readable medium of claim9 wherein the constructing step includes the step of parsing the inputstring to discern its syntactic and semantic structure.
 11. Thecomputer-readable medium of claim 9 wherein the identifying stepincludes the steps of:for each selected word of the inputtext:retrieving from a linguistic knowledge base one or more hypernymsof the selected word each having a similarity value characterizing thesimilarity in meaning of the hypernym to the selected word; andidentifying any of the hypernyms whose similarity value exceeds apreestablished threshold.
 12. The computer-readable medium of claim 9wherein the contents of the computer-readable medium further cause thecomputer system to perform the steps of:before the constructing step,selecting the input string from a search query; and submitting thegenerated tokens to a query engine for comparison to a representation ofone or more target documents.
 13. The computer-readable medium of claim9 wherein the contents of the computer-readable medium further cause thecomputer system to perform the steps of:before the constructing step,selecting the input string from a body of text to be indexed; andsubmitting the generated tokens to an indexing subsystem for storage inan index representing the body of text.
 14. The computer-readable mediumof claim 13 wherein the contents of the computer-readable medium furthercause the computer system to perform the step of determining an inversedocument frequency of each of the words occurring in the alternativelogical forms,and wherein the submitting step omits to submit to theindexing subsystem tokens representing alternative logical formscontaining words whose inverse document frequency is smaller than apreestablished minimum inverse document frequency.
 15. Thecomputer-readable medium of claim 13, further including the stepsof:after the submitting step, determining an inverse document frequencyof each of the words occurring in the alternative logical forms; andremoving from the index tokens representing alternative logical formscontaining words whose inverse document frequency is smaller than apreestablished minimum inverse document frequency.
 16. Thecomputer-readable medium of claim 9 wherein the identifying stepidentifies hypernyms of the selected words that have coherent hyponymsets with respect to the selected words.
 17. A method in a computersystem for creating and accessing an information retrieval index, theindex characterizing one or more indexed documents by storing tokenizedsemantic structures occurring in the indexed documents, the methodcomprising the steps of:for each of a plurality of passages in theindexed documents; receiving a plurality of tokenized semanticstructures all corresponding to the same passage, receiving an indexeddocument location for the passage, and storing in the index mappingsfrom each of the plurality of received tokenized semantic structures tothe index document location; and for a query issued against the indexeddocuments:receiving a tokenized semantic structure corresponding to thequery, identifying in the index a mapping from a tokenized semanticstructure matching the tokenized semantic structure corresponding to thequery to an identified indexed document location, and returning theidentified indexed document location.
 18. The method of claim 17 whereinthe step of receiving a tokenized semantic structure corresponding tothe query includes the step of receiving a plurality of tokenizedsemantic structures all corresponding to the same query, and wherein theidentifying step identifies a mapping from a tokenized semanticstructure matching any of the plurality of tokenized semantic structurescorresponding to the query.